{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6x1ypzczQCwy"
   },
   "source": [
    "# Creating a Vertex AI Feature Store Using the SDK\n",
    "\n",
    "## Learning objectives\n",
    "\n",
    "In this notebook, you learn how to:\n",
    "\n",
    "1. Create feature store, entity type, and feature resources. \n",
    "2. Import your features into Vertex AI Feature Store.\n",
    "3. Serve online prediction requests using the imported features.\n",
    "4. Access imported features in offline jobs, such as training jobs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tvgnzT1CKxrO"
   },
   "source": [
    "## Overview\n",
    "\n",
    "This notebook introduces Vertex AI Feature Store, a managed cloud service for machine learning engineers and data scientists to store, serve, manage and share machine learning features at a large scale.\n",
    "\n",
    "This notebook assumes that you understand basic Google Cloud concepts such as [Project](https://cloud.google.com/storage/docs/projects), [Storage](https://cloud.google.com/storage) and [Vertex AI](https://cloud.google.com/vertex-ai/docs). Some machine learning knowledge is also helpful but not required.\n",
    "\n",
    "### Dataset\n",
    "\n",
    "This notebook uses a movie recommendation dataset as an example throughout all the sessions. The task is to train a model to predict if a user is going to watch a movie and serve this model online.\n",
    "\n",
    "Each learning objective will correspond to a _#TODO_ in this student lab notebook -- try to complete this notebook first and then review the [solution notebook](../solutions/sdk-feature-store.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lWEdiXsJg0XY"
   },
   "source": [
    "## Before you begin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i7EUnXsZhAGF"
   },
   "source": [
    "### Install additional packages\n",
    "\n",
    "For this notebook, you need the Vertex SDK for Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "2b4ef9b72d43"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# The Google Cloud Notebook product has specific requirements\n",
    "IS_GOOGLE_CLOUD_NOTEBOOK = os.path.exists(\"/opt/deeplearning/metadata/env_version\")\n",
    "\n",
    "# Google Cloud Notebook requires dependencies to be installed with '--user'\n",
    "USER_FLAG = \"\"\n",
    "if IS_GOOGLE_CLOUD_NOTEBOOK:\n",
    "    USER_FLAG = \"--user\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "wyy5Lbnzg5fi"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: google-cloud-aiplatform in /opt/conda/lib/python3.7/site-packages (1.13.0)\n",
      "Requirement already satisfied: google-cloud-storage<3.0.0dev,>=1.32.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (2.3.0)\n",
      "Requirement already satisfied: google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (2.7.1)\n",
      "Requirement already satisfied: google-cloud-bigquery<3.0.0dev,>=1.15.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (2.34.3)\n",
      "Requirement already satisfied: packaging>=14.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (21.3)\n",
      "Requirement already satisfied: proto-plus>=1.15.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (1.20.3)\n",
      "Requirement already satisfied: google-cloud-resource-manager<3.0.0dev,>=1.3.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform) (1.4.1)\n",
      "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (2.27.1)\n",
      "Requirement already satisfied: protobuf>=3.12.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (3.20.1)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.52.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (1.56.1)\n",
      "Requirement already satisfied: google-auth<3.0dev,>=1.25.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (1.35.0)\n",
      "Requirement already satisfied: grpcio<2.0dev,>=1.33.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (1.46.1)\n",
      "Requirement already satisfied: grpcio-status<2.0dev,>=1.33.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (1.46.1)\n",
      "Requirement already satisfied: google-cloud-core<3.0.0dev,>=1.4.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.3.0)\n",
      "Requirement already satisfied: python-dateutil<3.0dev,>=2.7.2 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.8.2)\n",
      "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.3.2)\n",
      "Requirement already satisfied: grpc-google-iam-v1<0.13dev,>=0.12.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-resource-manager<3.0.0dev,>=1.3.3->google-cloud-aiplatform) (0.12.4)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging>=14.3->google-cloud-aiplatform) (3.0.9)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (0.2.7)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (4.2.4)\n",
      "Requirement already satisfied: six>=1.9.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (1.15.0)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (4.8)\n",
      "Requirement already satisfied: setuptools>=40.3.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (59.8.0)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (1.1.2)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (2021.10.8)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (3.3)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (2.0.12)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (1.26.9)\n",
      "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (1.15.0)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=1.25.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-aiplatform) (0.4.8)\n",
      "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.21)\n"
     ]
    }
   ],
   "source": [
    "# Install necessary dependencies\n",
    "! pip install {USER_FLAG} --upgrade google-cloud-aiplatform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hhq5zEbGg0XX"
   },
   "source": [
    "### Restart the kernel\n",
    "\n",
    "After you install the SDK, you need to restart the notebook kernel so it can find the packages. You can restart kernel from *Kernel -> Restart Kernel*, or running the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "EzrelQZ22IZj"
   },
   "outputs": [],
   "source": [
    "# Automatically restart kernel after installs\n",
    "import os\n",
    "\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    # Automatically restart kernel after installs\n",
    "    import IPython\n",
    "\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BF1j6f9HApxa"
   },
   "source": [
    "### Set up your Google Cloud project\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "1. [Enable the Vertex AI API and Compute Engine API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component).\n",
    "\n",
    "1. Enter your project ID in the cell below. Then run the cell to make sure the\n",
    "Cloud SDK uses the right project for all the commands in this notebook.\n",
    "\n",
    "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WReHDGG5g0XY"
   },
   "source": [
    "#### Set your project ID\n",
    "\n",
    "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "oM1iC_MfAts1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Project ID:  qwiklabs-gcp-01-5dbc4e7474d8\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "PROJECT_ID = \"qwiklabs-gcp-01-5dbc4e7474d8\"    # Replace this with your Project ID\n",
    "\n",
    "# Get your Google Cloud project ID from gcloud\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n",
    "    PROJECT_ID = shell_output[0]\n",
    "    print(\"Project ID: \", PROJECT_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qJYoRfYng0XZ"
   },
   "source": [
    "Otherwise, set your project ID here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "riG_qUokg0XZ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Project ID:  qwiklabs-gcp-01-5dbc4e7474d8\n"
     ]
    }
   ],
   "source": [
    "if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
    "    PROJECT_ID = \"qwiklabs-gcp-01-5dbc4e7474d8\"  # Replace this with your Project ID\n",
    "print(\"Project ID: \", PROJECT_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XoEqT2Y4DJmf"
   },
   "source": [
    "### Import libraries and define constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "Cdct_Lm7x2I_"
   },
   "outputs": [],
   "source": [
    "# Import necessary libraries and define required constants\n",
    "from google.cloud import aiplatform\n",
    "from google.cloud.aiplatform import Feature, Featurestore\n",
    "\n",
    "REGION = \"us-central1\"  # Replace this with your region \n",
    "if REGION == \"[your-region]\":\n",
    "    REGION = \"us-central1\"\n",
    "\n",
    "FEATURESTORE_ID = \"movie_prediction\"\n",
    "INPUT_CSV_FILE = \"gs://cloud-samples-data-us-central1/vertex-ai/feature-store/datasets/movie_prediction.csv\"\n",
    "ONLINE_STORE_FIXED_NODE_COUNT = 1\n",
    "\n",
    "aiplatform.init(project=PROJECT_ID, location=REGION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h_HmF24mBHv9"
   },
   "source": [
    "## Terminology and Concept\n",
    "\n",
    "### Featurestore Data model\n",
    "\n",
    "Vertex AI Feature Store organizes data with the following 3 important hierarchical concepts:\n",
    "```\n",
    "Featurestore -> Entity type -> Feature\n",
    "```\n",
    "* **Featurestore**: the place to store your features\n",
    "* **Entity type**: under a Featurestore, an Entity type describes an object to be modeled, real one or virtual one.\n",
    "* **Feature**: under an Entity type, a Feature describes an attribute of the Entity type\n",
    "\n",
    "In the movie prediction example, you will create a featurestore called `movie_prediction`. This store has 2 entity types: `users` and `movies`. The `users` entity type has the `age`, `gender`, and `liked_genres` features. The `movies` entity type has the `titles`, `genres`, and `average rating` features.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9UvxYyGUimKw"
   },
   "source": [
    "## Create Featurestore and Define Schemas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "buQBIv3ZL3A0"
   },
   "source": [
    "### Create Featurestore\n",
    "\n",
    "The method to create a Featurestore returns a\n",
    "[long-running operation](https://google.aip.dev/151) (LRO). An LRO starts an asynchronous job. LROs are returned for other API\n",
    "methods too, such as updating or deleting a featurestore. Running the code cell will create a featurestore and print the process log."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "D6uIWQeoBSr8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating Featurestore\n",
      "Create Featurestore backing LRO: projects/555331480095/locations/us-central1/featurestores/movie_prediction/operations/240926705865195520\n",
      "Featurestore created. Resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction\n",
      "To use this Featurestore in another session:\n",
      "featurestore = aiplatform.Featurestore('projects/555331480095/locations/us-central1/featurestores/movie_prediction')\n"
     ]
    }
   ],
   "source": [
    "# Create the Featurestore\n",
    "fs = # TODO 1a: Your code goes here(\n",
    "    featurestore_id=FEATURESTORE_ID,\n",
    "    online_store_fixed_node_count=ONLINE_STORE_FIXED_NODE_COUNT,\n",
    "    project=PROJECT_ID,\n",
    "    location=REGION,\n",
    "    sync=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ag8pCQ7rNjVf"
   },
   "source": [
    "Use the function call below to retrieve a Featurestore and check that it has been created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "eKhD4q8rXjmM"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name: \"projects/555331480095/locations/us-central1/featurestores/movie_prediction\"\n",
      "create_time {\n",
      "  seconds: 1653293620\n",
      "  nanos: 221462000\n",
      "}\n",
      "update_time {\n",
      "  seconds: 1653293620\n",
      "  nanos: 282563000\n",
      "}\n",
      "etag: \"AMEw9yPvXZbOmobQ3MMAa8p4y5diQ8WcP0R7ryxPbgIgv0aVf5RDiSzPm5D3GFiVOHtb\"\n",
      "online_serving_config {\n",
      "  fixed_node_count: 1\n",
      "}\n",
      "state: STABLE\n",
      "\n"
     ]
    }
   ],
   "source": [
    "fs = Featurestore(\n",
    "    featurestore_name=FEATURESTORE_ID,\n",
    "    project=PROJECT_ID,\n",
    "    location=REGION,\n",
    ")\n",
    "print(fs.gca_resource)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EpmJq75zXjmT"
   },
   "source": [
    "### Create Entity Type\n",
    "\n",
    "Entity types can be created within the Featurestore class. Below, create the Users entity type and Movies entity type. A process log will be printed out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "GU0oXvINBgPV"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating EntityType\n",
      "Create EntityType backing LRO: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users/operations/5028253109760032768\n",
      "EntityType created. Resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users\n",
      "To use this EntityType in another session:\n",
      "entity_type = aiplatform.EntityType('projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users')\n"
     ]
    }
   ],
   "source": [
    "# Create users entity type\n",
    "users_entity_type = fs.create_entity_type(\n",
    "    entity_type_id=\"users\",\n",
    "    description=\"Users entity\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "qPCGFznrFwFy"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating EntityType\n",
      "Create EntityType backing LRO: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/movies/operations/2701018002316328960\n",
      "EntityType created. Resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/movies\n",
      "To use this EntityType in another session:\n",
      "entity_type = aiplatform.EntityType('projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/movies')\n"
     ]
    }
   ],
   "source": [
    "# Create movies entity type\n",
    "movies_entity_type = fs.create_entity_type(\n",
    "    entity_type_id=\"movies\",\n",
    "    description=\"Movies entity\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G0TS9i5SJnkt"
   },
   "source": [
    "To retrieve an entity type or check that it has been created use the [get_entity_type](https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/featurestore/featurestore.py#L106) or [list_entity_types](https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/featurestore/featurestore.py#L278) methods on the Featurestore object.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "5DHcAyoFJmq9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<google.cloud.aiplatform.featurestore.entity_type.EntityType object at 0x7f74f4e2cb90> \n",
      "resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users\n",
      "<google.cloud.aiplatform.featurestore.entity_type.EntityType object at 0x7f74f69c9a10> \n",
      "resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/movies\n"
     ]
    }
   ],
   "source": [
    "users_entity_type = fs.get_entity_type(entity_type_id=\"users\")\n",
    "movies_entity_type = fs.get_entity_type(entity_type_id=\"movies\")\n",
    "print(users_entity_type)\n",
    "print(movies_entity_type)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "EWMzDNFT5qf0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<google.cloud.aiplatform.featurestore.entity_type.EntityType object at 0x7f74f50c7750> \n",
       " resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/movies,\n",
       " <google.cloud.aiplatform.featurestore.entity_type.EntityType object at 0x7f74f4e17310> \n",
       " resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fs.list_entity_types()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FJW4q-0jO2Xf"
   },
   "source": [
    "### Create Feature\n",
    "Features can be created within each entity type. Add defining features to the Users entity type and Movies entity type by using the `create_feature` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "PvjwT84iVSps"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating Feature\n",
      "Create Feature backing LRO: projects/555331480095/locations/us-central1/featurestores/movie_prediction/operations/2695388502782115840\n",
      "Feature created. Resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users/features/age\n",
      "To use this Feature in another session:\n",
      "feature = aiplatform.Feature('projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users/features/age')\n",
      "Creating Feature\n",
      "Create Feature backing LRO: projects/555331480095/locations/us-central1/featurestores/movie_prediction/operations/7870024474630815744\n",
      "Feature created. Resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users/features/gender\n",
      "To use this Feature in another session:\n",
      "feature = aiplatform.Feature('projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users/features/gender')\n",
      "Creating Feature\n",
      "Create Feature backing LRO: projects/555331480095/locations/us-central1/featurestores/movie_prediction/operations/2722410100546338816\n",
      "Feature created. Resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users/features/liked_genres\n",
      "To use this Feature in another session:\n",
      "feature = aiplatform.Feature('projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users/features/liked_genres')\n"
     ]
    }
   ],
   "source": [
    "# to create features one at a time use\n",
    "users_feature_age = users_entity_type.create_feature(\n",
    "    feature_id=\"age\",\n",
    "    value_type=\"INT64\",\n",
    "    description=\"User age\",\n",
    ")\n",
    "\n",
    "users_feature_gender = users_entity_type.create_feature(\n",
    "    feature_id=\"gender\",\n",
    "    value_type=\"STRING\",\n",
    "    description=\"User gender\",\n",
    ")\n",
    "\n",
    "users_feature_liked_genres = users_entity_type.create_feature(\n",
    "    feature_id=\"liked_genres\",\n",
    "    value_type=\"STRING_ARRAY\",\n",
    "    description=\"An array of genres this user liked\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RQ9-AyFYBvcX"
   },
   "source": [
    "Use the [list_features](https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/featurestore/entity_type.py#L349) method to list all the features of a given entity type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "S5Lr8uXk1xD0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<google.cloud.aiplatform.featurestore.feature.Feature object at 0x7f74f4e42b90> \n",
       " resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users/features/liked_genres,\n",
       " <google.cloud.aiplatform.featurestore.feature.Feature object at 0x7f74f69c9fd0> \n",
       " resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users/features/gender,\n",
       " <google.cloud.aiplatform.featurestore.feature.Feature object at 0x7f74f4e330d0> \n",
       " resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users/features/age]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_entity_type.list_features()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "llTT9_Dgbac2"
   },
   "outputs": [],
   "source": [
    "movies_feature_configs = {\n",
    "    \"title\": {\n",
    "        \"value_type\": \"STRING\",\n",
    "        \"description\": \"The title of the movie\",\n",
    "    },\n",
    "    \"genres\": {\n",
    "        \"value_type\": \"STRING\",\n",
    "        \"description\": \"The genre of the movie\",\n",
    "    },\n",
    "    \"average_rating\": {\n",
    "        \"value_type\": \"DOUBLE\",\n",
    "        \"description\": \"The average rating for the movie, range is [1.0-5.0]\",\n",
    "    },\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "YhfOKJL_BvuM"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch creating features EntityType entityType: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/movies\n",
      "Batch create Features EntityType entityType backing LRO: projects/555331480095/locations/us-central1/featurestores/movie_prediction/operations/1569488595939491840\n",
      "EntityType entityType Batch created features. Resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/movies\n"
     ]
    }
   ],
   "source": [
    "# Create Features\n",
    "movie_features = # TODO 1b: Your code goes here(\n",
    "    feature_configs=movies_feature_configs,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xUVOzrAb1AFX"
   },
   "source": [
    "## Search created features\n",
    "\n",
    "While the `list_features` method allows you to easily view all features of a single\n",
    "entity type, the [search](https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/featurestore/feature.py#L352) method in the Feature class searches across all featurestores and entity types in a given location (such as `us-central1`), and returns a list of features. This can help you discover features that were created by someone else.\n",
    "\n",
    "You can query based on feature properties including feature ID, entity type ID, and feature description. You can also limit results by filtering on a specific featurestore, feature value type, and/or labels. Some search examples are shown below. \n",
    "\n",
    "Search for all features within a featurestore with the code snippet below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "OAiv8p8LMdLJ"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<google.cloud.aiplatform.featurestore.feature.Feature object at 0x7f74f50c7c90> \n",
       " resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/movies/features/average_rating,\n",
       " <google.cloud.aiplatform.featurestore.feature.Feature object at 0x7f74f4e2cbd0> \n",
       " resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/movies/features/genres,\n",
       " <google.cloud.aiplatform.featurestore.feature.Feature object at 0x7f74f4e17590> \n",
       " resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/movies/features/title,\n",
       " <google.cloud.aiplatform.featurestore.feature.Feature object at 0x7f74f4e4e310> \n",
       " resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users/features/age,\n",
       " <google.cloud.aiplatform.featurestore.feature.Feature object at 0x7f74f4b29210> \n",
       " resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users/features/gender,\n",
       " <google.cloud.aiplatform.featurestore.feature.Feature object at 0x7f74f69ced90> \n",
       " resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users/features/liked_genres]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_features = Feature.search(query=\"featurestore_id={}\".format(FEATURESTORE_ID))\n",
    "my_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jcxsiBUiIyvE"
   },
   "source": [
    "Now, narrow down the search to features that are of type `DOUBLE`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "a9ovJSyEI4OZ"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name: \"projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/movies/features/average_rating\"\n",
       "description: \"The average rating for the movie, range is [1.0-5.0]\"\n",
       "value_type: DOUBLE\n",
       "create_time {\n",
       "  seconds: 1653294299\n",
       "  nanos: 993065000\n",
       "}\n",
       "update_time {\n",
       "  seconds: 1653294299\n",
       "  nanos: 993065000\n",
       "}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "double_features = Feature.search(\n",
    "    query=\"value_type=DOUBLE AND featurestore_id={}\".format(FEATURESTORE_ID)\n",
    ")\n",
    "double_features[0].gca_resource"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wtr9tvH6JAOY"
   },
   "source": [
    "Or, limit the search results to features with specific keywords in their ID and type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "3G1mNV1uJFBC"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name: \"projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/movies/features/title\"\n",
       "description: \"The title of the movie\"\n",
       "value_type: STRING\n",
       "create_time {\n",
       "  seconds: 1653294299\n",
       "  nanos: 989948000\n",
       "}\n",
       "update_time {\n",
       "  seconds: 1653294299\n",
       "  nanos: 989948000\n",
       "}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "title_features = Feature.search(\n",
    "    query=\"feature_id:title AND value_type=STRING AND featurestore_id={}\".format(\n",
    "        FEATURESTORE_ID\n",
    "    )\n",
    ")\n",
    "title_features[0].gca_resource"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K3n5XdK8Xjmw"
   },
   "source": [
    "## Import Feature Values\n",
    "\n",
    "You need to import feature values before you can use them for online/offline serving. In this step, you learn how to import feature values by ingesting the values from GCS (Google Cloud Storage). You can also import feature values from BigQuery or a Pandas dataframe.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BlqJ-QdTcs6W"
   },
   "source": [
    "### Source Data Format and Layout\n",
    "\n",
    "BigQuery table/Avro/CSV are supported as input data types. No matter what format you are using, each imported entity *must* have an ID; also, each entity can *optionally* have a timestamp, specifying when the feature values are generated. This notebook uses Avro as an input, located at this public [bucket](https://console.cloud.google.com/storage/browser/cloud-samples-data-us-central1/vertex-ai/feature-store/datasets). The Avro schemas are as follows:\n",
    "\n",
    "**For the Users entity**:\n",
    "```\n",
    "schema = {\n",
    "  \"type\": \"record\",\n",
    "  \"name\": \"User\",\n",
    "  \"fields\": [\n",
    "      {\n",
    "       \"name\":\"user_id\",\n",
    "       \"type\":[\"null\",\"string\"]\n",
    "      },\n",
    "      {\n",
    "       \"name\":\"age\",\n",
    "       \"type\":[\"null\",\"long\"]\n",
    "      },\n",
    "      {\n",
    "       \"name\":\"gender\",\n",
    "       \"type\":[\"null\",\"string\"]\n",
    "      },\n",
    "      {\n",
    "       \"name\":\"liked_genres\",\n",
    "       \"type\":{\"type\":\"array\",\"items\":\"string\"}\n",
    "      },\n",
    "      {\n",
    "       \"name\":\"update_time\",\n",
    "       \"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-micros\"}]\n",
    "      },\n",
    "  ]\n",
    " }\n",
    "```\n",
    "\n",
    "**For the Movies entity**:\n",
    "```\n",
    "schema = {\n",
    " \"type\": \"record\",\n",
    " \"name\": \"Movie\",\n",
    " \"fields\": [\n",
    "     {\n",
    "      \"name\":\"movie_id\",\n",
    "      \"type\":[\"null\",\"string\"]\n",
    "     },\n",
    "     {\n",
    "      \"name\":\"average_rating\",\n",
    "      \"type\":[\"null\",\"double\"]\n",
    "     },\n",
    "     {\n",
    "      \"name\":\"title\",\n",
    "      \"type\":[\"null\",\"string\"]\n",
    "     },\n",
    "     {\n",
    "      \"name\":\"genres\",\n",
    "      \"type\":[\"null\",\"string\"]\n",
    "     },\n",
    "     {\n",
    "      \"name\":\"update_time\",\n",
    "      \"type\":[\"null\",{\"type\":\"long\",\"logicalType\":\"timestamp-micros\"}]\n",
    "     },\n",
    " ]\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "m7DyDa6chbJx"
   },
   "source": [
    "### Import feature values for Users entity type\n",
    "\n",
    "When importing, specify the following in your request:\n",
    "\n",
    "*   IDs of the features to import \n",
    "*   Data source URI\n",
    "*   Data source format: BigQuery Table/Avro/CSV\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "dS8QbDPfkSEe"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['liked_genres', 'gender', 'age']\n"
     ]
    }
   ],
   "source": [
    "# Specify the required details of users entity\n",
    "USERS_FEATURES_IDS = [feature.name for feature in users_entity_type.list_features()]\n",
    "USERS_FEATURE_TIME = \"update_time\"\n",
    "USERS_ENTITY_ID_FIELD = \"user_id\"\n",
    "USERS_GCS_SOURCE_URI = (\n",
    "    \"gs://cloud-samples-data-us-central1/vertex-ai/feature-store/datasets/users.avro\"\n",
    ")\n",
    "GCS_SOURCE_TYPE = \"avro\"\n",
    "WORKER_COUNT = 1\n",
    "print(USERS_FEATURES_IDS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "x86Dtg0Vu-uf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Importing EntityType feature values: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<google.cloud.aiplatform.featurestore.entity_type.EntityType object at 0x7f74f4e2cb90> \n",
       "resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Import EntityType feature values backing LRO: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users/operations/5577692264299233280\n"
     ]
    }
   ],
   "source": [
    "# Import feature values for the users entity type\n",
    "users_entity_type.ingest_from_gcs(\n",
    "    feature_ids=USERS_FEATURES_IDS,\n",
    "    feature_time=USERS_FEATURE_TIME,\n",
    "    entity_id_field=USERS_ENTITY_ID_FIELD,\n",
    "    gcs_source_uris=USERS_GCS_SOURCE_URI,\n",
    "    gcs_source_type=GCS_SOURCE_TYPE,\n",
    "    worker_count=WORKER_COUNT,\n",
    "    sync=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "laXdJPIqkLJO"
   },
   "source": [
    "### Import feature values for Movies entity type\n",
    "\n",
    "Similarly, import feature values for the Movies entity type into the featurestore.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "-c_CHjd11AD0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['genres', 'title', 'average_rating']\n"
     ]
    }
   ],
   "source": [
    "# Specify the required details of movies entity\n",
    "MOVIES_FEATURES_IDS = [feature.name for feature in movies_entity_type.list_features()]\n",
    "MOVIES_FEATURE_TIME = \"update_time\"\n",
    "MOVIES_ENTITY_ID_FIELD = \"movie_id\"\n",
    "MOVIES_GCS_SOURCE_URI = (\n",
    "    \"gs://cloud-samples-data-us-central1/vertex-ai/feature-store/datasets/movies.avro\"\n",
    ")\n",
    "GCS_SOURCE_TYPE = \"avro\"\n",
    "WORKER_COUNT = 1\n",
    "print(MOVIES_FEATURES_IDS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "Q4mMBRTc1Xpx"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Importing EntityType feature values: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/movies\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<google.cloud.aiplatform.featurestore.entity_type.EntityType object at 0x7f74f69c9a10> \n",
       "resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/movies"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Import EntityType feature values backing LRO: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/movies/operations/6834196560335601664\n"
     ]
    }
   ],
   "source": [
    "# Import feature values for the Movies entity type\n",
    "# TODO 2: Your code goes here(\n",
    "    feature_ids=MOVIES_FEATURES_IDS,\n",
    "    feature_time=MOVIES_FEATURE_TIME,\n",
    "    entity_id_field=MOVIES_ENTITY_ID_FIELD,\n",
    "    gcs_source_uris=MOVIES_GCS_SOURCE_URI,\n",
    "    gcs_source_type=GCS_SOURCE_TYPE,\n",
    "    worker_count=WORKER_COUNT,\n",
    "    sync=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9TdxPYdDXjnA"
   },
   "source": [
    "## Get online predictions from your model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ezJIMyU-XjnB"
   },
   "source": [
    "[Online serving](https://cloud.google.com/vertex-ai/docs/featurestore/serving-online)\n",
    "lets you serve feature values for small batches of entities. It's designed for latency-sensitive service, such as online model prediction. For example, for a movie service, you might want to quickly show movies that the current user would most likely watch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "foNB0D2aw37c"
   },
   "source": [
    "### Read one entity per request\n",
    "\n",
    "With the Python SDK, it is easy to read feature values of one entity. By default, the SDK will return the  latest value of each feature, meaning the feature values with the most recent  timestamp.\n",
    "\n",
    "To read feature values, specify the entity type ID and features to read. By default all the features of an entity type will be selected. The response will output and display the selected entity type ID and the selected feature values as a Pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "084G80L_DWwd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EntityType feature values imported. Resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/movies\n",
      "EntityType feature values imported. Resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction/entityTypes/users\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>entity_id</th>\n",
       "      <th>age</th>\n",
       "      <th>liked_genres</th>\n",
       "      <th>gender</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bob</td>\n",
       "      <td>35</td>\n",
       "      <td>[Action, Adventure]</td>\n",
       "      <td>Male</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  entity_id  age         liked_genres gender\n",
       "0       bob   35  [Action, Adventure]   Male"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read feature value of user entity by using entity ID\n",
    "users_entity_type.read(entity_ids=\"bob\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "qrR-SY3i58rh"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>entity_id</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>movie_01</td>\n",
       "      <td>The Shawshank Redemption</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  entity_id                     title\n",
       "0  movie_01  The Shawshank Redemption"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read feature value of movies entity by specifying the entity type ID and features ID\n",
    "# TODO 3: Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oYk83Zt9xF8m"
   },
   "source": [
    "### Read multiple entities per request\n",
    "\n",
    "To read feature values from multiple entities, specify the different entity type IDs. By default all the features of an entity type will be selected. Note that fetching only a small number of entities is recommended when using this SDK due to its latency-sensitive nature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "BIJFcIIHULOd"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>entity_id</th>\n",
       "      <th>gender</th>\n",
       "      <th>liked_genres</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>alice</td>\n",
       "      <td>Female</td>\n",
       "      <td>[Drama]</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>bob</td>\n",
       "      <td>Male</td>\n",
       "      <td>[Action, Adventure]</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  entity_id  gender         liked_genres  age\n",
       "0     alice  Female              [Drama]   55\n",
       "1       bob    Male  [Action, Adventure]   35"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_entity_type.read(entity_ids=[\"bob\", \"alice\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "DtL0uy_0Dcca"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>entity_id</th>\n",
       "      <th>title, genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>movie_02</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>movie_03</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>movie_04</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  entity_id title, genres\n",
       "0  movie_02          None\n",
       "1  movie_03          None\n",
       "2  movie_04          None"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies_entity_type.read(\n",
    "    entity_ids=[\"movie_02\", \"movie_03\", \"movie_04\"], feature_ids=[\"title, genres\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Sds42j8ZsCzS"
   },
   "source": [
    "Now that you have learned how to fetch imported feature values for online serving, the next step is learning how to use imported feature values for offline use cases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WpvhPAYxD-Ml"
   },
   "source": [
    "## Get batch predictions from your model\n",
    "\n",
    "Batch serving is used to fetch a large batch of feature values for high-throughput, and is typically used for training a model or batch prediction. In this section, you learn how to prepare for training examples by using the Featurestore's batch serve function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tPzAGvqJHh3B"
   },
   "source": [
    "### Use case\n",
    "\n",
    "**The task** is to prepare a training dataset to train a model, which predicts if a given user will watch a given movie. To achieve this, you need 2 sets of input:\n",
    "\n",
    "*   Features: you already imported into the featurestore.\n",
    "*   Labels: the ground-truth data recorded that user X has watched movie Y.\n",
    "\n",
    "\n",
    "To be more specific, the ground-truth observation is described in Table 1 and the desired training dataset is described in Table 2. Each row in Table 2 is a result of joining the imported feature values from Vertex AI Feature Store according to the entity IDs and timestamps in Table 1. In this example,  the `age`, `gender` and `liked_genres` features from `users` and\n",
    "the `titles`, `genres` and `average_rating` features from `movies` are chosen to train the model. Note that only positive examples are shown in these 2 tables, i.e., you can imagine there is a label column whose values are all `True`.\n",
    "\n",
    "[batch_serve_to_bq](https://github.com/googleapis/python-aiplatform/blob/main/google/cloud/aiplatform/featurestore/featurestore.py#L770) takes Table 1 as\n",
    "input, joins all required feature values from the featurestore, and returns Table 2 for training.\n",
    "\n",
    "<h4 align=\"center\">Table 1. Ground-truth data</h4>\n",
    "\n",
    "users | movies | timestamp            \n",
    "----- | -------- | -------------------- \n",
    "alice  | Cinema Paradiso     | 2019-11-01T00:00:00Z \n",
    "bob  | The Shining     | 2019-11-15T18:09:43Z \n",
    "...   | ...      | ...     \n",
    "\n",
    "\n",
    "<h4 align=\"center\">Table 2. Expected training data generated by using batch serve</h4>\n",
    "\n",
    "timestamp            | entity_type_users | age | gender | liked_genres | entity_type_movies | title | genre | average_rating  \n",
    "-------------------- | ----------------- | --------------- | ---------------- | -------------------- | - | -------- | --------- | ----- \n",
    "2019-11-01T00:00:00Z | bob              | 35        | M                | [Action, Crime]      | movie_02            | The Shining | Horror | 4.8 \n",
    "2019-11-01T00:00:00Z | alice             | 55        | F                | [Drama, Comedy]      | movie_03          | Cinema Paradiso | Romance | 4.5 | \n",
    "... | ... | ... | ... | ... | ... | ... | ... | ...\n",
    "             "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hKwWGg2i4a4G"
   },
   "source": [
    "#### Why timestamp?\n",
    "\n",
    "Note that there is a `timestamp` column in Table 2. This indicates the time when the ground-truth was observed. This is to avoid data inconsistency.\n",
    "\n",
    "For example, the 2nd row of Table 2 indicates that user `alice` watched movie `Cinema Paradiso` on `2019-11-01T00:00:00Z`. The featurestore keeps feature values for all timestamps but fetches feature values *only* at the given timestamp during batch serving. On that day, Alice might have been 54 years old, but now Alice might be 56; featurestore returns `age=54` as Alice's age, instead of `age=56`, because that is the value of the feature at the observation time. Similarly, other features might be time-variant as well, such as `liked_genres`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gAL9Y4VTOLT1"
   },
   "source": [
    "### Create BigQuery dataset for output\n",
    "\n",
    "You need a BigQuery dataset to host the output data in `us-central1`. Input the name of the dataset you want to create and specify the name of the table you want to store the output created later. These will be used in the next section.\n",
    "\n",
    "**Make sure that the table name does NOT already exist**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "4Oc-jrd6Ow7N"
   },
   "outputs": [],
   "source": [
    "# Import necessary libraries\n",
    "from datetime import datetime\n",
    "\n",
    "from google.cloud import bigquery"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "QOaGi2PrEAwA"
   },
   "outputs": [],
   "source": [
    "# Output dataset\n",
    "DESTINATION_DATA_SET = \"movie_predictions\"  # @param {type:\"string\"}\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
    "DESTINATION_DATA_SET = \"{prefix}_{timestamp}\".format(\n",
    "    prefix=DESTINATION_DATA_SET, timestamp=TIMESTAMP\n",
    ")\n",
    "\n",
    "# Output table. Make sure that the table does NOT already exist; the BatchReadFeatureValues API cannot overwrite an existing table\n",
    "DESTINATION_TABLE_NAME = \"training_data\"  # @param {type:\"string\"}\n",
    "\n",
    "DESTINATION_PATTERN = \"bq://{project}.{dataset}.{table}\"\n",
    "DESTINATION_TABLE_URI = DESTINATION_PATTERN.format(\n",
    "    project=PROJECT_ID, dataset=DESTINATION_DATA_SET, table=DESTINATION_TABLE_NAME\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "RKhmymT-O0vy"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created dataset qwiklabs-gcp-01-5dbc4e7474d8.movie_predictions_20220523083542\n"
     ]
    }
   ],
   "source": [
    "# Create dataset\n",
    "client = bigquery.Client(project=PROJECT_ID)\n",
    "dataset_id = \"{}.{}\".format(client.project, DESTINATION_DATA_SET)\n",
    "dataset = bigquery.Dataset(dataset_id)\n",
    "dataset.location = REGION\n",
    "dataset = client.create_dataset(dataset)\n",
    "print(\"Created dataset {}.{}\".format(client.project, dataset.dataset_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W8dLJ9nuDFgI"
   },
   "source": [
    "### Batch Read Feature Values\n",
    "\n",
    "Assemble the request which specify the following info:\n",
    "\n",
    "*   Where is the label data, i.e., Table 1.\n",
    "*   Which features are read, i.e., the column names in Table 2.\n",
    "\n",
    "The output is stored in the BigQuery table.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "0p752VDHSmFn"
   },
   "outputs": [],
   "source": [
    "SERVING_FEATURE_IDS = {\n",
    "    # to choose all the features use 'entity_type_id: ['*']'\n",
    "    \"users\": [\"age\", \"gender\", \"liked_genres\"],\n",
    "    \"movies\": [\"title\", \"average_rating\", \"genres\"],\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "VsAaLGyHRYll"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Serving Featurestore feature values: projects/555331480095/locations/us-central1/featurestores/movie_prediction\n",
      "Serve Featurestore feature values backing LRO: projects/555331480095/locations/us-central1/featurestores/movie_prediction/operations/155358312945156096\n",
      "Featurestore feature values served. Resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<google.cloud.aiplatform.featurestore.featurestore.Featurestore object at 0x7f74f4b41690> \n",
       "resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Batch read the feature values\n",
    "# TODO 4: Your code goes here(\n",
    "    bq_destination_output_uri=DESTINATION_TABLE_URI,\n",
    "    serving_feature_ids=SERVING_FEATURE_IDS,\n",
    "    read_instances_uri=INPUT_CSV_FILE,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3RAdjahHQ93J"
   },
   "source": [
    "After the LRO finishes, you should be able to see the result in the [BigQuery console](https://console.cloud.google.com/bigquery), as a new table under the BigQuery dataset created earlier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TpV-iwP9qw9c"
   },
   "source": [
    "## Cleaning up\n",
    "\n",
    "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
    "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
    "\n",
    "You can also keep the project but delete the featurestore and the BigQuery dataset by running the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "NBTNfN8vxz4x"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting Featurestore : projects/555331480095/locations/us-central1/featurestores/movie_prediction\n",
      "Delete Featurestore  backing LRO: projects/555331480095/locations/us-central1/operations/7078516840120451072\n",
      "Featurestore deleted. . Resource name: projects/555331480095/locations/us-central1/featurestores/movie_prediction\n"
     ]
    }
   ],
   "source": [
    "# Delete Featurestore\n",
    "fs.delete(force=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "kzkD2SWJ2gIK"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleted dataset 'movie_predictions_20220523083542'.\n"
     ]
    }
   ],
   "source": [
    "# Delete BigQuery dataset\n",
    "client = bigquery.Client(project=PROJECT_ID)\n",
    "client.delete_dataset(\n",
    "    DESTINATION_DATA_SET, delete_contents=True, not_found_ok=True\n",
    ")  # Make an API request.\n",
    "\n",
    "print(\"Deleted dataset '{}'.\".format(DESTINATION_DATA_SET))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "BlqJ-QdTcs6W",
    "hKwWGg2i4a4G"
   ],
   "name": "sdk-feature-store.ipynb",
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-6.m92",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m92"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
